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ABSTRACT . ' r . • , 

A needs assessment was- conducted to determine the 
degree to which 'Pittsburgh Public Schools were meeting the 1 
educational needs of their stydents.. Different groups were surveyed 
to determine their perceptions of conditions in the district. Data 
collection and analysis established the status of children's- 
educatiotialrneedfe. The fiistrict's most serious problems were: 
improving- si«dent .achievement in £asic skills; implementing 
procedures for per^nnel evaluation; attracting and maintaining 
students - ; managing the impact of enrollment decline: to control costs 
and increas'e -quality; and developing a 'strategy for individual school 
improvement. The results were disseminated through interactive slide 
show sessions and written reports. Utilization of needs assessment 
results are of concern to etfaj.ua to'r<£\ Three' categories of use are 
instrumental, conceptual, and persuasive. Factors 'influencing 
\ ^utilization are timing; ^identification* of a primary client; 
evaluation design; avoidanc'e of political attacks; variety of 
researqh strategies; substantive knowledge of education as a process 
v and structure; preliminary testing of sjurvey instruments; atmosphere 
for presentation of results; fcn<| the credibility of data. The authors 
recommend a, client orientation approach for school district 
evaluation. (DWH) > ^ % * 
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In his excellent review of *needs assessments in educational 

evaluation, Stufflebeajn (J1977) points out t^iat he found actual examples 

( . * - 

of needs assessments to be "a much better scmrce of operational guidance 

for doing needs assessments than was the thebrejtipal literature of 

evaluation" '(p. ' 10). , Similarly, Alki'o, Daillak and White t(1979). ■ 

demonstrate tfie value of case studies in contributing to • an 

-a * * * 

understanding of evaluation utilization . 

It is # i'n that spirit that we relate this e^erience about the 
conduct .of a n£ed$ assessment done for the Pittsburgh Public Schools. 
This paper summarizes Sow the assessment was conducted and how it was 
used. We think it worth tellirfe .because it may be useful to other , 
districts that are contemplating doing a district-wide needs assessment,, 
and also because it has implications for improving, the utility of school 
district evaluation efforts. * # 



The .reseafch reported herein was supported by the Learning Research and 
Development Center ^ supported in part by funds from the National 
Institute of Education. (NIE), United' States Department of Education. 
The opinions, expressed do not 'necessarily reflect the position or policy 
of NIE, and no official endorsement should be inferred. 



Background 



First, it "is necessary to provide a description of the context for 

this work. The' Learning Research" and Development Center- (LRDC) is *a 

* • 

university-based R&D center that receives njost of its funding from the 
National Institute, of Education. Within LRD.C*,. the ^valuation Research < 
Unites working oawaysta improve evaluation- research. This . £rpup 
represents fixture ^f full and part* -time 'facult^, ^graduaie swdehts,' 
J aria - 'dtfpa^'t >ta£f . ' !; f ' r - , \ # 



^ One';<jf the questions being^'addresfeed by .the Evaluation Unit is how 
jto improve' 7 district-based evaluatibn .studies so that the results of 
io.cal research ef foirts .can be used to improve -local "school practices. 
-We haye chosen t£ work, on school district evaluation by d6ing it, as 
opposed to observing, or surveying what existing district-based' offices 
are doing. This is possible 'because the Pittsburgh Public Schools, the 

. * i 9 

school district £n which IRDC is located, dotes not have an evaluati^ft 

* * * » 

office. ^n establishing a Working relationship with the Dis-trict, the- 

Evaluation Unit Vained a " site for further understanding evaluation 

'reWarch^* Th^ District - gained an evaluation research capability that. 

wou<ld otherwise be beyond its current means.; 

There are currently 45,t)00 students enrolled in. the Pittsburgh 

/"..<•• 

Public Schools, which ' represents • a steady decline from the early 

* * i * - 0 

V % • 

seventies when over 75,000 '.students were enrolled in the District. 
Policy is set by*A*"hine member, elected school board irt cooperation with 
$he superintendent of schools and a central administrative .bureaucracy. 

There are approximately 3 % ,200 teachers arid support staff on the pay?oll. 

ft " * 

Like most Northeastern, urban districts, Pittsburgh faces a familiar set 
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Of problems that include declining enrollment, rising costsV shrinking 

tax bases and rising demands. v 

, • \ * 

Over 'the two years prior .to .the Fall of 1980, we , had developed a 

* \ • 

good working* relationship with the District;*, building up the trust and 

. " \ 

.confidence thajt this kind of\effort requires-. .Most of oar projects with 
distript '.administrators were specific program evaluations/- In th6. 
Spring of 1980, however', things -became ■ a little "untidy-,. The Mstridt's 
latest* desegregation *plan was rejected b"^ the '.Peftnsylvania t Human 
Relations Commission; * the' 'School Board * chose npt -to renew the 

■ '. * .•■■ >■ - % • . t ' • . 

Superintendent's contract; and the teacher's union announced that they 

• - * : . . v 4 x . ' 

would go on strike in the fall if* their demands were not met. Of 

course/ these are the* kinds of events that urban districts are 

* 

♦experiencing all the time. Eventually ' the teacher's contract was 

* r *► * 

settled, a desgregat'ion -pl*an was approved, and^the School Board selected 

* . 1 * . y 

Richard Wallace as their new Superintendent. * 

■ : ■ 

IS The needs assessment story begins on September 10, 1980 when we met 
with the t\ew superintendent and offered the continued assistance of the 
Evaluation Unit. During our first meeting, he indicate^ that it- would 
be very important to conduct a district-wide needfc assessment that* would 
enable him to t establish priorities for the District. Several, subsequent 

jneetings with the Superintendent were held to define "the general nature, 

: * • 

purpose and' procedures to be used for' this assessment- 

The primary purpose of the assessment was to, determine « the degree 
to which the Pittsburgh Public Schools were meeting* the educational 

needs of the children in the District, ani^do it in**a ; way. wl^ich would 

• . * * * * 

suggest priorities for improving the educational program in the 
District. THe general objective^ Was to ' identify . conditions in the 
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, District that can and should be improved. 



Components ^of the Neejis Assessment 



Two approaches werer used inVhis assessment/ Qne approach* was to 
survey different, groups regarding their perceptions of current 

• conditions in the District* The groups sampled included administrators, 

* * 
board members,' counselors, custodians, nurses, 'paraprof essionals , 

parents, -principals and other building administrators, psychologists, 

gchool secretaries; security guards, social workers, students, 
* • « 

supervisors, teachers,* and other Pittsburgh citizens... A task force 
consisting of representatives of many of these different group^ was, 
established to assist in the desigp of the surveys. Over 1000 people 
participated in ( the surveys. . , . . 

' * fhe other approach "to the assessment involved collecting and 

analyzing existing school district data in order to establish the 

i * 
current status of the educational, needs . of the children. , Such , data 

included student achievement, failure and dropout rates, attendance, 

f ' ' ' < ' 

suspensions, and demographic descriptions of *the students in each^ 
school, as ,well as data describing the resources available*in the % 

> .school, such as the number of teachers. The"resulting data base covered 
the past five years. •"- This ma_d£ it possible to examine trends over time 

. as veil as- relation^hips^among the many*di£ ferent variable? in any given 



. year. 
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Results 
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% 

The results were organized in terms of five broad areas that were 
* t ' ^ 

found to be 'the, District's'most pressing needs Although some groups 

* 

ranked them differently than others, thjere was consensus that these are 
the District's most pressing problems: (1) improve student achievement 
in basic skills, (2) implement , procedures for personnel evaluation, (3) 

attract and hold students, (4) manage the impact of enrollment decline 

t # 

o • * 

in order to hold down unwanted costs and increase quality whei^e it is 
needed, and (5) develop a strategy for improving individual schools. 

The more specific survey results and the five-year data base were 
used to further define the nature of each of the*se problem areas, and 
possible strategies* for dealing with them. .The student achievement 
issue illustrates this approach. • • - 

i 

Need to Improve Student Achievements * 

The improvement of student achievement was identified as the number 
one problem in th£ District. by most survey respondents. This fact was 
not surprising since previous reports of standardized test results -in 
the District had received riuch attention. The. achievement problem that 
hfid been previously highlighted was that students began to fall behind 
national ^'norms ^at about 'the^'fourth grade. The phrase "fourth 'grade 
slump" was commonly heard during Board deliberations o£ achievement 
issues* • 
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•AjS part* of this needs assessment, additional) analyses were 
conducted to determine possible reasorfs for . this decline. A major 
achievement -problem -identified in the analyses was that a large number 
of children leave • third grade still not reading well enough to use 

• reading in subsequent, learning. Thfe test results indicated that 
approximately "25 percent. *of the students completed third grade with 

^inadequate reading comprehension'skills . It was* also 'noted , that ' this 
percentage, varied considerably among schools. That is, in .some schools 
as many as 50 percent of the children did taot comprehend what they read 
as they entered, fourth gratf%, whereas in other schools, all of the 
children leaned "to be reading well enough* to 'begitf^dUrt^ grade . 



In an attempt to improve achievement, the District offers many 

remedial programs 'for children who are not learning ^from normal 

< * * 

classroom instruction. These programs .include Title I, Project -Pass, 
* * ■ • 

special education programs for the mildly handicapped (e.g*, ID and EMR) 
• £nd the 12th», grade remedial program. Such* programs have become, so 
extensive in the District that in some gtade levels, 5& percent of the 
children are participating in one or more of these remedial efforts. As 

part of the assessment, the current 'structure of remedial programs was 

-i • » 

' investigated, and the results indicated that lack of« coordination among 
* ■ l' ' • 

the various remedial effoirts was a serious problem* 

• • / i 

• An examination of the achievement levels of the student© in these 

remedial programs* suggested * fthat thefee different programs are serving 

' students with similar achievement problems • 9 Thus, children may be 

- p 

labeled and isolated for special treatments in spite of the fact that 
childten with different ^labels appear to t?e havfng. similar -academic 
'difficulties and aye receiving similar treatments urfder, these different 
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labels. Tha~t is, different programs were trying to work dri similar 



problems, often with ^he same- general strategies, but the teachers and 
supervisors involved were not benefitting from eacK other's experiences. 
Different supervisory st^ff were assigned to the different programs, 
even in the same building. , r 

Another need was to improve the coordination between what" tias # 
taught in the remedial program and what was taught; in the student's 
'regular classroom. Teachers, principals and supervisors all* recognized 
• fchis as a critical task for improving remedial instruction. This 
assessments-suggested the need to integrate the remedial services insofar 
as possible, reduce the use of stigmatizing labels for these children, 
and put the focus on the early grades so that students develop the basic 
skills before they move into the intermediate grades. 

Other Needs, identified 4 ^ 

* 

In addition to achievement, the other four major problem areas were 
examined using both survey and existing District data. A summary report 
is available from the authors ("A Needs Assessment of ..the Pittsburgh 
Public Schools**), but it is useful to provide a brief description of the 
other" results in this. paper. 

1 ' 
Personnel evaluation was >one of the most consistently mentioned 

i problems by the people who v work in the District- Few seemed happy with 

present procedures. Employees wanted their evaluation oriented toward 

^improving ' job 'performance, 'and, felt it was important to clarify who is 

•%> ♦ 

' « \ _ " 

responsible for evaluating whom and how those evaluations should be 
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\ * Another, serious problem facing the Pittsburgh' schools is the 

increasing proportion, of children going elsewhere. for fheir education* 

i _ • 

It seemed to many that this was threatening thQ health and vitality of 
the public schools, which have been and* should continue to be a critical 



ingredient in this democracy. The major reason people gave for avoiding 

' -\ ■ ■ ' ' 

*- ■ 

the public schools was thpt student discipline is a serious problem. 
Teachers and principals agreed that there was a lack of alteratives for 
dealing with discipline problems and a need to consistently enforce 
policies. 

The heed to manage enrollment decline also emerged ks a . priority 

- " . / 
area. On a per pupil basis, costs were going up faster than one would 

expect from inflation. This was primarily because fixed building .costs 

- V 

were being spread over fewer students in each school, and class sizes 
were getting smaller.. Although smaller classes can be an important 
component v of a strategy, t for improving the quality of classroom 
instruction > the .reduction in class size was very uneven and did *not^ 
occur where it was needed. The assessment concluded that the money^ 
saved by closing additional schcTols can be used to increase the quality 
of the educational program where it is most needed *~ 

The needs assessment also suggested -focusing upon individual 
school^ because most, of t;He identified needs must be attacked at the 
building level^ and because some schools are clearly in greater need of. 

improvement than others. An index of school descriptors was developed* 

*• * 

that indicated wher6 to begin a focused schoql improvement effort • It 
was suggested that a detailed diagnosis of the needs orf the identified 
sdnools could be the basis for designihg improvement strategies in each 
school. • 



Page 9 



Dissemination . 



Dissemiftatiori of thefce results took place between January 13* and 
» * 

March 5, ^1981. Two approaches, were used, interactive slide show 

sessions and written reports. The former proved to be especially useful 

< / - , 

in both getting the message* acrosjs, as well ^.sharpening the message 

* 7 

th^bugh helpful comments offered by the audiences. The schedule was as 
follows: 



Superintendent 



January 13 ^ 



Superintendent and School Board President January 20 

The Board of Education ' January 24 v 

Central Administration * 1 ^ . February 12 

Building Administrators *and Supervisors February 24 

Teachers ^ ^ * * February 25 

• Task Force ' fc • "March 3 v 

Press Conference * March 5 



Each , dissemination session involved l'ooking at the results, with 
overhead slides .and handouts, in an interactive mode. This was 
particularly true of- the meeting with the nine-member Soard/for whom a 



/ 



on 



epday retreat to a nearby -col^A^e campus was arranged • [ For the 3000 
teachers, however, we had to settle fbi a one-hour TV presentation which 
they viewed in their tniildings. This was followed by a discussion of 
the results among the > faculty a^d building administrators in each. 



school* 



/ 



The written reports werfr kept short." \ We prepared a two-page 

abstract, with the* level/ of* detail illustrated on page 7, a'16-p^e 

/ * * . 
* • / 

summary detailed to the degree «of the achievement discussion, and six 

special topic papers, ' running ^about ten/ pages each, on theme's of 

particular irftere^t: /achievement, discipline, communications, personnel 
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evaluation, organizational* structure* arid school improvement. We also 
prepared a .memo «for the Superintendent; *«which summarized' specific 
"suggestions to the new- superintendent" Jthat emerged f rom\ the surveys t>f 
the various groups. * • 

* Utilization. 



This section presents a» discussion of what i!s meant by utilization, 

followed by a/ description of some of the ways in which t;he needs 
* « * * * • 

* . * • 

assessment has been utilized in the months "following its dissemination. 

* ' % 

We ' th^n go on *to, examine' the factors that seemed to influence the 
» ° • . • 

"utilization that has, occurred', and then briefly relate this experience 
to the emerging literature on utilization. 1 



A Definition of - Utilizatipn 

As r-esearchers have become more and pior^ concerned about the extent 
to which ♦evaluation results are being utilized, . there has. been a 
concomitant intejrefet determining what constitutes use. Part of .the 
reason for the interest in defining use stems from the fact that many 
researchers were reaching the conclusion that evaluation research was 
Used' very little in policy formation and decision-making. (Worthen & 
SanderS, 1973; Rossi, 1972; Dexter, 1066; Guba, 1969; Mann, 1972; 
Cohen and Garet, 1975; Lynn, 1978; Cook, 1978). 



v 
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1 Some researchers % unwilling to accept the rather blegk picture of 

utilization, argued that the definition of use being applied' was too 

narrow. A definition of use founded on the'notion that a major ^policy 

decision Would turn on the production of a single piece of evaluation 

• • * 

research was hopelessly disadvantaged in any attempt to find instances 
of* use. This was* thought to be so 'f or two reasons. Firs^, such a 
definition was naive about the policy process itself. Decision-makers, 
reach a decision through afi of ten -amorphous -set of minor decisions,* 
meetings, and. interaction's with constitutencies . , Weiss (1980) describes 

this process as "decision by accretion. " Second, a concentration on the^ 

\ 

traditional ^definition had the effect of actually screening the many 
• * * 

different uses that actually occur , at 'jiiore modest antf mundane levels as 

V ' . 

a decision accretion process takes placet. * , 

Proponents of a brqader view of; utilization include Weiss 1979; 

Alkin,. et al», 1979; and Pkt^o^, 1978.* Weiss (1*979) has offerfed seven 

categories of use, each bound to particular . aspects of the 

decision-making process. Patto^ sug^est^ ttiat* looking for utilization 

• - # *- * , 

requires that t you look for gradual changes in .the policy .process or 

* \ * , ' • / 

decision-maker's viewpoint over time, where m a piece* of evaluation 

information is one of a. number of data dQnsidered^ by the decision-maker. 

Alkin et al. ' (1979) ..write -that use occurs when,' apaong other things, 

^ — » . ^ . * t 

"evaluation information. . '.[is]. . .used.* .,. . in. making decisions,"; 
substantiating 'decisions or actions or altering^attitudes" (p. 226/. 

Although we agree with those writers that view use as occurring . In, 

'many forms, in this report we have ^chosen to describe relatively 

straightforward examples of use that would, in all likelihood ,^f it most 
j 

traditional cle&riitions. This* was-, done for two reasons. First, the 
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fact that direct use 'has occurred is inland of itself 'important "to 
record. Second,^ finding good methods for systematically tracking and 
documenting th§ more subtle uses that are occurring is a problem that is < 
being worked on but is by no means satisfactorily resolved at^this time'; 
It is hoped 'that as more data are collected, more can be said about 
Subtle uses of evaluation research in subsequent papers. « > 



Among the variety of writers who, have examined the issue of what' 
constitutes utilization, the review of research by Leviton and Hughes 

(1981) is particularly helpful. In this synthg^A^ they identify three 

f • > 

broad categories of use: " instrumental , Conceptual and persuasive" 
(p. 4); Briefly, instrumental use occurs when specific instances could 
be documented of ways in which research was being used for 
decision-making or problem-solving purposes. Conceptual use occurs when 
a policy maker's , £hinking about a particular issue is influenced by a 

*** * ' * 

piece of evaluation information. Persuasive use occurs when evaluation 
information is used to support" "or t defend a particular position or 
policy. Using these, three categories of use as a - basic structure, 
examples of how district- decision-makers used the fesultS'-of tfre needs 
assessment fire reported in the next ^sectipn. * < ' 



Examples of Use ' K * ' . - * ■ . * 

t ■ .. : . ;\v ' ; - * ' 

The first indication that the- assessment was going to be used came 
in the farm of^a *jnews .release on the Monday following the weekend, 
retreat with the, Superintendent and Board. when the- preliminary^, results 
of the needs ' assessment *were described in detail. Here are important 
excerpts .from that release. , 
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"Wallace Announces Plans for School Improvement" 



Pittsburgh, fa.,- January 26, 1981 . . . • A 

Superintendent of^ Schools Richard C. Wallace today announced* 
that he will immediately begin to develop "action plans*' for 
th§ 'School District that -focus on ^two ma^jor areas, school >, 
imprdvemement and cost effective management. . # • 

* 9 

The decision to focus on the two major areas was made jointly 
by Wallace and members of the Board of Education after they 
had the bpportunity to review preliminary results of the 
districtwide needs assessment initiated by the superintendent 
shortly after he took office in the Fall. • . \ 

"This needs assessment information combined with other 
* educational data, such' as an analysis of test* scores over 1 the % 
past five years, helped us establish the priorities," , Wallace 
said. "Once we °develop^the action plans, all of us in the 
District can channel our energies iji the same direction and 
respond to concerns identified by the groups of people the 
school serve." 



On February 25, the Board passed the resolution that- adopted school 

improvement and cost effective management as the top two priorities for 

* » " 

the District. In that resolution, the Board agreed to "assign resources 

and concentrate its efforts* to resolve problems within these prioiFity 

areas, in # relation to: 

1 

1. Student achievement in the basic skills 

2. Staff evaluation 

3. Student disoipline ' 

4. Attracting and holding students 9 

5. Enrollment decline ^ 

"6. Increasing the effectiveness of individual schools." 



The resolution made it clear th$t the two priority^ areas and the six 

# » 
specific" needs-. surfaced as major concerns as the result of the district- 
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V 

wide needs assessment; "Five of the' six priority areas, were stated in 
language identical tQ that used in the assessment. The^d'Sition of 
discipline as a separate p'riority was a modification that .occurred 
during *the dissemination process as early drafts had included this issue 
\under the priority of attr^acfting and*htflding . students. More will be 
said about how and why this change came about *in Q tjie following* section, 

v ^ As indicated in the January press release, the Superintendent set* 
in -motion the development of action plans for each i>rit>rity area. Task 
forces established for this purpose consisted of representatives of 
various segments of the District and each group worked toward a July 15 
deadline, at whl/ch time detailed action plans were' pres£ftted to the 
Board as a follpw-up to their February 25 ^resolution. The Boa*d was 
given tw^pponths td review the action plans, and on ^September- 23, one 
'year from the initiation of the assessment, they were formally adopted, 



The presentation by the. Superintendent to the Board of the results 
of the* assessment and the passage by the Bbard'of the priority areas 
identified in the plan is a straightforward. example of what Leviton and 
Hughes (1981) have described as instrumental use of evaluation 
information. Research Was commissioned to identify current conditions 
in need of improvement -in the .District. 'Summary reports were prepared 
and presentations were made. On the basis of these data, the Board 
resolved to attack the identified problems through the development of 
specific action plans and through the allocatfon of District resources 
to support thege plans. 



• 1 
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♦ • 

The needs assessment process also provides an excellent example of 

v ♦ - " 

* .*? 

-"what -Levi ton and Hughes (1981) have called conceptual use, ways in which 
evaluation information has influenced the way decision-makers think 

o 

about "a problem. Identification of conceptual use is not contingent, 
upon a piece ofcssfevaluation information directly influencing a specific 
decision, program' or issue. The influence of "the study on achievement 
undertaken as part of the needs assessment is an example of^ conceptual 
utilization. • 



As noted earlier, achievement reports prepared by the District's 
Testing Office had identified a decline in achievement>starting at about 
the fourth grade and gradually increasing through the middle school 
year9>. The term "fourth gr'^de slump" hacT become a common" term used 
among Board members during the past several years 'to describe the 
achievement issue in thje District. Also, remedial programs in the 
District were expanding into the middle and upper grades' in an attempt 
at dealing with th§t "slump. 

The needs assessment more closely examined the achievement levels 
being attained by District students in the primary (1-3) grades. 
District managers were shown that twenty-five petcent of the students 
were bringing serious reading problems into the fourth grade and, that 
the root of the problem lay with the instruction being received in the 
early years. The net effect of this research has been to begin to 
refocus the attention of Distict managers upon instruction in the 
primary grades as a way o^ solving the decline in test scores in the 
district that begins in the* fourth grade. 
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Thr^nefcds a.sses.sment process ,* as distinguished frjom the ^peds 
assessment ^results, provides- c an exai^ple of utilizat^0tf for persuasive 



provides- c an exai^ple of utilizatJjD'ti 



purposes. This has occurred at a variety of levels in the District and 
in' a number »of ways. ; The ^persuasive use in the context of 



Superintendent-Board relations is illustrative* 



etfious 



When the new Superintendent took office in the FalL of 1980, he was 
confronted with * a nine member, elected Board that had a recent history 
of sharp division and fragmentation. Th^primary issue of contention 
among B&ard members had been over plans for districts-wide desegregation. 
The termination of the previous superintendent\had , in large measure, 
been the* result of a failure' to work-out a solid majority on the Board 
for any single desegregati^T strategy. Fortunately for the^ newly 
arriving Superintendent, the desegregation .issue per se had temporarily 
fceeti relegated to the "baqk burner" f or ^^e Board. ^T^^neasy and 
fragile coalition of six Board members had passed a pl^ 
summer. Whilfe litigation challenging the plan was underway, therd was a 
general consensus on the Board that everyone should pull together to 
make the opening of the school year as quiet and reassuring to a nervous 
public . as possible. There also seemed to.t&e a feeling of • exhaustion 
among many Board members over the acrimony that had attended so many of 
the discussions on desegregation., There was a willingness to let the 
courts (or the elections to come in 1982) stimulate new moves. 

Arriving with a knowledge bf this context, the new Superintendent 
seemed ? to have aenseti the time was ripe for some new initiatives in the 
District. As an • "outsider" (the previous three superintendents had come 
up through the system) he had tffie advantage of being able to take a 
fresh look at the issues ♦ He also was not bound by previous 
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relationships and commitments likely to be built up duping* a career-long < 

r ' 4 * 

rise within a single district. Yet, the problem remained of how to 

develop some consensus for action among bpard members that had been so • 

( ftV 

recently divided. The development of a "needs assessment, especially 
that cqmgonent* of the assessment that* surveyed dfcl of the major 
constituencies of the District, was an important mechanism for building 
a consensus for action an^ong Board members. Issues identified through 
an assessment process clearly could not be attributed to the special 
interest of one or the other of the Board factions. As such, .the 
^assessment ^provided a "neutral ground", in effect, for building a new 
consensus among key decision-makers in the District- Similar persuasive > 
uses of the assessment results could be noted in the- Superintendent's*' 



relationships with his. own central administrative staff. The point her 
* is simply that^the assessment process did indeed fulfill a persuasiv^ 
IS use as defined by Leviton and Hughes (1981). 



Although examples of instrumental, conceptual, and persuasive 
utilization have been treated as discrete entities, it is not tbe intent 
here (nor is it the intent of Leviton and Hughes) to leave the 
impression that these are necessarily mutually exclusive categories. 
For example, the achievement study cited as having conceptual use may 

^eventually also turn-out to have an instrumental use r to the extent that, 
over time, programs, ' policies*, and resources are shifted as a result of 

. the realization that the fourth grade slump can be reduced by improving 
primary grade instruction. Nevettheless, the three categories are. 
helpful in describing the complex influences evaluation information can* 
have. -In the following section, discussion is provided that, considers 
some Q#"£he features of this assessment that may have contributed to its> 

- W •--*'•' 

.utility. Although this is highly speculative on our i>art, it seems 
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important to reflect' on the utilization question so that this experience 
and others like it might begin to suggest procedures for school district 
evaluations that will increase their potential for improving educational 
programs r ^ \ 

Factors That Influenced Utilization « . ' 



Turning to the factors that may, have influenced use of the needs 
assessment, it should be noted that no single factor or set of factors 
seemed* to be sufficient in guaranteeing that the evaluation information 
would be utilized.. For District evaluators considering the question of 
use, this experience suggests that one should, think in terms t of 
increasing the probability of use rather than looking for the key 
technique or strategy! that would insure utilization. 

, Timing . The needs assessment seemed to be the right task at the 
right ^ime. • It has already been noted that the decision by fhe 
Superintendent tq call for an assessment was heavily influenced by th^ 
context of recent Board division over the desegregation issue.. For his 
new administration, * the assessment provided a new start for new 
, initiatives,, and most importantly a "neutral" ground upon which some 
Board consensus could be constructed that would not be diminished by the 
acrimony characteristic^ of the previous -months. The Board timbers 
•seemed somewhat relieved by* the opportunity for a "freph.. start" in their 
own deliberations. The importance of timing has b.een noted by 'some 
researchers (e.g.^Hill, 1980; Patton, 1978), and questioned by others 
(e.g., King & Thompson, 1981). ■ . m , 
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A client was clearly identified . The Superintendent was the 
'primary client and tfrfe evaluation tfea^worked closely with him during 
the design, ^analysis and dissemination phases of the needs assessment. 
While the ultimate users of the result's ranged widely across the 
District from the School Board to individual program or school managers, 
it was felt to be extremely important that a single client- guided the*, 
basic structure of the^fc^aluation. Given the political climate in the 
District, to have attempted to explicitly serve multiple clients might 
easily have* led to producing evaluation results th^at would not be usable 
by any group. , Hill (1980$ has a useful analysis, in the context of 
Congressional evaluation research, of the importance of identifying 
one's client to facilitate utilization. 

-r 

• It shodld also be noted, however ; - that while the single client 

approach can .clearly and positively influence use, ? it canalso create 

problems for a district evaluator. Keeping to a single client ^was not 

an easily managed task. Just to give one example, when instruments to 

be used in the survey were piloted among several task forces 

representing the various constituencies to be polled, there was strong 
a * * * 

pressure from individuals representing middle management (principrfLs) to* 

keep questions about. . how they- , should ' be ' evaluated off of the 

instruments. The Superintendent, as primary client, was the final 

« ♦ it 
arbiter and inpisteB that the evaluation question remain (although ixf 

'^modified form) in the survey. As i£ turned out,* evaluation of personnel 

surfaced as one of the six priorities identified by the'District. 
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t - * Another example concerns^ agproblem- th^fc is .ethical in nature. „When 

does -serving a client to insure use', begin to impinge upon the 

, ' *\ ' ' ' < 

responsibility of the evaluator Which is presumably to teport results as 

forthrightly as possible? 'This is ' a fine line} especially hard to 

distinguish in that .it'might be crossed 3s ./easily ^tfirough acts bf^ 

omission as through any gross "massagfflg" of data* . In the needg 

assessment experience there was a double* advantage , of , '(1) having a 

client who was quitfe willing to "let the chips fall where .they may 11 and, 
► • * 

(2) having the evaluation research conducted by. individuals outside of 

the- District power structure and thus, presumably, less vulnerable to 

• # * unwarranted influence. As Crohbach (1980) notes- in another context , "the 

' • best that can be said on. this issue of the client-evaluat!or relationship 

v c , is that ;fc the crucial ethical problem appears to be freedom 4 to 

m h * * o * 

, communicate duririgi and after: the study," subject to the legitimate 

* * « « ' T 

concerns for privacy^ national security, and faithfulness to contractual 
commitments" (p 6).^ .» . 

# * \ ™ ' 9 . * . • \ 

Nature of the evaluation ^design * The design %f the needs 

assessment was established in a way that', in • itself, encouraged 

" - / ' ':' ; - " \. 

utilization at^various levels in' the District;* For >exainple, summaries 
of the surveys -of ? various constituencies were prepared tha.t> not only 
" * identified problem areas,: but al^o developed credible ways of 

establishing .which areas were most important' and v ^foX what groups. In 
addition, the survey- elicited -possible solutions -to Sfftecific problem 
areas* Thus, .information, linking specific problems to a range of 

7 s ' alternative solutions ?was available to decision-makers giving them both * % 

* - •* - . . - . ■ • ' * . •••..«• 

• - ' * * • . - " ' s • ' 

^<>~ sensfe'/of ' the key- issues and .also ideas about 'how^tp start, attacking 

^-'7;' ^ : -v.. -7"^- % - * 



. * - . 1'** • a, - 



Personal characteffi|ffics of- the jase;£ . \The Superintendent 2s the 
primary client Had a st^^^T^^ing thaf 0 the fl research was implemented 
and used. A number of 'authqrs, ;!jaJFe* noted the importance of user' 
characteristics in influencing v us,e o£e^;/ Crotibach, 19,80j / Kennedy, 
Apling, Neumann, 1980). In- thisrc^f § j>ithe' Sifperintendent clearly had a 

large stake in using the infomatidrv-fo^^uildrng a consensus for action 

r: . • 

among his various constituencies ^in-jhke, District. He also happened to 
have a strong research background tfiat made 'him both comfortable with 
evaluation research methodology as' well, as committed to using research 
to help inform the policy .proceaj?* Both of these factors insured the 



on-going involvement o'f^ the primary client. 4t "Night noted that this 
interest on the part of the Superintendent was especially crucial to the. 
data collection activities in the analytical ,>phase . of the assessments 
Access to data was greatly facilitated in fhe District's bureaucracy 
because of the visible importance attached s ,to the activity by ^the 

OS „ - . ■ 

Superintendent. r . ' 

v *. * • * Hp 

. • . ; ' ' V. 

Avoidance of political land'mlpes; Information, was summarized in 

it*. ; >: . •• • 

ways that remained true to what respondents A were attempting to 
communicate while at • the same" time avoiding personal attacks on 
individuals. Once'said, this point' is .perhaps obvious to all. However, 
given that a number of questions* ^ on thfe purvey were open-ended in 
nature, the possibility (and, as it turned out, the a fact) that some of 
the data could be construed as persona?^ attacks-- made it important to 
summarize* the points without '.the '. invective. t This is justified on 
founds that a public airing of data at a ^per'sonaX .level might well 
infiame tlje dissemination process to sue H an extent as .ta hinder this 
larger purpo.se- of the assessment.. Rerhaps more importantly, * such 
discretion is warranted in , a situation whete^as is, the case of an 
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* v anonymous survey .respondent , the "accused** cannot behold and challenge 
the accuser. * 

Variety of research strategies used , ihe combined use of surveys 
an<f detailed- analyses o%*. existing data 'bases contributed . to the 
probability that the results of the assessment would be used. (See ^ 
Cro'nbach, 1980.) The survey data provided important data about the r 
^perceptions of key constituencies. This information was especially (but 
not exclusively.) useful as^a way of gauging the political needs and 
stress points in the District as far as potential, prbgramy initiatives 
were concerned. Yet, the identification of problem areas was clearly, 
not enough to lead decision-makers to a program of action. 

In some instances, problems were identified Jbut the solution, 
proposed wa£ pither ambiguous or" altogether absent. Thte achievement 
issue is a good case in ppint. Almost all survey respondents- recognized 
that there was a problem.* However, the specific nature of the problem, 
as well as the examination oft what to do about it, could be best done by- ,, 
looking at some long-term, trends in District data and programming. The 
* point, is that in this experience no single research strategy was* 
sufficient to fully address the client's interests, and, the questions 
being addressed dictated the methodological strategies employed rather 

• :* 

than the reverse. 

* - 
technology versus substantive knowledge . .The experience of' this 

case study suggests that in district evaluation research, elegant 
^ applications of sophisticated analytical technologies can be less? 
+ important* in influencing* utilization than the employment of a sound 

knowledge or educational structure and the polities that influenw^it. 

# While some -relatively sophisticated methodologies were^eftployed in the 

• • 23,, * . ... \^ ' 
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analysis of existing data bases, the results produced would " have been 
less interpretable, and cer'tainly less intelligible to decision-makers, # 
without an indepth' understanding x>f how specific programs, ^schools and 
classes were ^organized in the District/ This is not to suggest that- 
sophisticated statistical techniques cannot be relevant, but rather, 
that; these are^only useful to the extent that they fit the question and 
that the results produced ca£ be interpreted iit the context of the 
District's educational realities. Perhaps' the most powerful "metric*.* 

provided in the entire assessment process was the "twenty-five percent" 

9 

figure of students leaving third grade not reading well enough to learn 
from subsequent reading. In a related point, it should also be noted 
that research jargon is rarely' useful in communicating with 
decision-makers. In every report apd presentation, attempts were made 
to delete such language from the research t^o be reported.* 

The issue- of technology versus substantive knowledge raises the 
corollary issue of what might be the best background and training 
strategies for preparing individuals to ^work as district evaluators. 
Lyon,. Doscher, McGranahan, and Williams (1978) report that about 
forty-two percent of those surveyed did not have a background in 
teaching. Many have backgrounds in Hferd sciences. The Pittsburgh 
experience suggests that some background, or at least understanding- qf 
education as» & process and structure, is useful in helping an evaluator 
frame the issues. for research. * 

¥ • { 

Pilot testing the instruments . J The research 'design benefitted from 
pilot testing the survey instruments with representatives of 
constituencies to be surveyed. Questions, were # invariably Sharpened, 
dropped or added as a result of interactions with task forces set up for 
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this purpose. It should be noted, -however, that in testing instruments 
there - is a chance t*hat unwanted shaping -of the evaluation can occur. 
Certainly the pressure exerted on the research team to eliminate 
personnel evaluation questions wag an example of how edrly testing of 

the instrument fean embroil the fevaluation in controversy even before the 

• * 

first stages of ,the research are completed. While recognizing this, 
.danger, on balance, the experience in Pittsburgh suggests that pilot 
testing can greatly improve the research design by sharpening the focjus 

9 

of the questions and that this can do .much towards 'increasing the 
relevance of the final results. ' % 

How you report it; can make a. difference . The research' team chose 
to use an 4 Interactive style as a primary mechanism for reporting 
research results; . One example of this was the all day retreat arranged 
by the Superintendent for the Board ^ The entire agenda consisted of the- 

reporting and discussion of the preliminary results of /the assessment. 

* * » 

The presentation to the Board was tried out ahead of time in several 
private forums (one meeting with colleagues at the research center, onfe 
with the Superintendent ' and another witli the Superintendent and Board. 
President) to wox£ on ttih structure of the program and the 
intell^ibility of the 'results. • 



The actual presentation to the Board occurred at a college campus 
away from the District's administrative .offices and' 4 local tnedia 
representatives that regularly "cover Board meetings., With distractions 
thus kept at minimum and with an entire day set aside for discussion, 
the' presentations by the Superintendent and the evaluation team took, 
place in an atmosphere that %ended to maximize dialogue. One example of 
how this setting and style of dissemination influenced use through^ the 



involvement of the. audience can be seen in the emergence of a sixth 
* 

priority from the needs assessment (i»e., discipline) in the final 
report. Although discipline had been included as part of the original 
message, it ,was subsumed by- the evaluation team under the "attracting 
and* holding 'students'* J.ssue. ' The Board clearly fe'lt that this? issue 
deserved to be highlighted as x a separate issue giv£n the strength of the 

, results presented- Final evaluation- .reports - reflected t this 

1 ■ . • \ •* 

reorganization. The interaction between the evaluators and potential 
users in this instance^provided an opportunity to' tecast the results in 
ways that increased the "investment'* ^users had in the results. 

* One 'fiVial, comment on" reporting strategies concerns the style of * 
reports offered as v back-up to the numerous presentations that were m&de. 
Four different- reports 'were prepared. Each report varied as to content, 
details and format in relation, to' the prospective audience and potential 
use, to be made of the information by that* audience. Fox • example, a 
brief .^summary of highlights was prepared " for use with media.* Result's 
and Distriat # .foHoW-up plans % were stressed. A sixteen-page .summary of 
results organized by priority* issue was prepared • for. use in Board 
deliberations • Wore detailed and lengthy analyses of \ survey data by 

- , ' 

issue', but^not confined 'to the feix priority areas, Were written for 
administrative personnel interested in specific problems and solutions 
identified by respondents . Finally, specific suggestions from the 

survey for th^riew Superintendent were prepared- as a memo to him. With 

V * ? - • * " 

* * fc ■ *. 

the exception of the memo to the 'Superintendent, all reports were made 
available to the public apd to. constituencies' in the public. This 

approach allowed individuals' to seek the^level 4 of Retail that they might 

* * • * . 

want for their own purposes without necessarily overburdening tg.ach 

reader with a k mountain of information if he/she was simply seeking a 
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quick summary of. the results. The utility of an approach that seeks to 
vary reporting and dissemination strategies with th.e make-up of 
potential usfers has been noted elsewhere in the literature (e ; g., Dkt£p t 
1979, Sproull & Larkey, 1979). • . . * ^ 

Credibility of the data ♦ Sottfe of the findings "rang true" for some 
* * * * 

potential users, cotlf inning their own experiences. - Some of the findings 

wfcre surprising to potential users but the evidence offered was both 

understandable and convincing. In each of these cases, the credibility 

of the data clearly influenced District decision-makers in their 

♦ * 
willingnes*s to *u§e • the evaluation information* An example of 

information confirming existing beliefs was the finding * that discipline 

was of primary importance to patents sending their children to private 

rather than public schools ♦ The Superintendent and individual members 

^of the Board had personal experiences^ttiat confirmed ^his finding ♦ 

The analyses of achievement and the "fourth grade slump** was t an 

V 

example of potential users, being surprised by a finding but, ultimately, 

> v 

• r 

being convinced by the evidence 4 .that was presented* --The- statistic 
mentioned earlier *^that appoximately twenty-five percent $f the. third 
graders lfeave that grade unable to read well enough to learn from 
subsequent ' reading, seemed to have a powerful impact oif the thinking of 
District decision-makers* ' ♦ 
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Concluding Comments 

* • 

Thfe. factors discussed in the previous^ section are offered with the 

* » 

caveats that no single variabl€^£nsures utilization and that any 

evaluator attempting to improve use' should .thifik in terms of increased 

probabilities, varying * strategies with the context and type of 

e^luation. As our^work in school district evaluation continuel|| there < 

will be further opportunities to confirm or question the relative 

strengths of some of the variables noted above* However, having said 

that no single strategy is sufficent, we would like to briefly return to 
I 

one variable, client orientation, that, if not-sufficient, is at least 
the place to start in improving the probabilities of .use in school 

district research* t f ; 

• ^ ' "* 

r * 

Most ^valuation research 1 * in a school district will have the 



potential for multiple user, aud lends* In stressing that the 
Superintendent wa^J jie primary client of the needs assessment, the 
emphasis wdt be on primary and not exclusive. The public, the Board., 
district managers were all clearly secondary clients and potential us x e^s-' 
of the "information. The importance to the evaluator of having a primary 
client lies in his/her ability to sufficiently define the questions to 
be addressed to avoid what Tukey (as quoted in Raiffa, 1968) has called 
Type til error: not asking 'the right question. It is our impression 
.that identifying a primary client and establishing a frank dialogue with 
that client through both evffuation design and implementation phases of 
the^, w6rk greatly enhances the chances that the results of an evaluation 
wlll.be used. » ' 

0 r 




In suggesting a client orientation approach for school district 
evaluation, it is recognized ^ th^t^ certain problems can -occur. The 
ethical question discussed in a previous section is one example. A 
second problem lies in the fact* that strong client-evaluator 
communication is very often not part 6f the tradition of school district 
^relationships. . A recent survey (King &, Thompson ^ 1981) of the 
perceptions about evaluators held by school 'boards and school 
administrators indicated that "few LEA users speak frequently with 
program evaluators about the programs they are studying" (p. 9,). Support 
for similar findings are to be found in other studies (e.g., David 1981; 
Lyon, Doscher, McGran^hap, & Williams, 1978). -< Achieving communication 
between district decision-makers and evaluators secg&s to be an important 
first T step In improving the use of evaluation research**" in school 
districts. * V 
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